Internal Notes for Reviewers

sc-8468

Validator Fundamentals

REGISTRATION PAGE PREVIEW — 🚨 SEE COURSE MODULE PREVIEW DISCLAIMER BELOW!!!! 🚨

• Like the Developer Fundamentals, I updated this course to be much more comprehensive yet straightforward.
• There are now updated versions of the login instructions, individual module learning outcomes, module summaries, additional slides for things not previously covered, etc.
• I thoroughly tested & timed these courses to make sure the instructions and the estimates made sense.

Course modules

🚨 NO LIVE PREVIEW 🚨 — AS THE IFRAME EMBEDS PULL FROM THE ROOT OF THE DOCS-DEMO FOLDER, YOU'LL NEED TO PULL DOWN THIS PR TO REVIEW & TEST THE TRAINING AS THE ONLINE HOSTED VERSION WON'T DRAW FROM THE CORRECT NOTEBOOK FILES:

1. Retrieve the branch & load the preview:

gh pr checkout 713
cd site
quarto preview

2. Navigate to Training > Validator Fundamentals.

Notebook execution

Proof that the development and validation execute notebooks both work even if the nested PR iframe can't display them properly:

Makefile

Adjusted the Makefile to simply copy over the notebooks/tutorials/ folder into a notebooks/EXECUTED/ directory including the model_validation courses:

Also updated the backup make execute command:

Now, the command asks you for which profile you want, which set of notebooks you want executed (or a custom path), and then asks you for your credentials (as you need separate developer/validator credentials to execute each set of notebooks without errors) to create a temporary .env file:

❯ make execute
Select PROFILE:
1) exe-demo
2) exe-staging
3) exe-prod
Enter choice [1-3] (default: 1): 1
Select FILE_PATH:
1) notebooks/EXECUTED/model_development
2) notebooks/EXECUTED/model_validation
3) Enter custom filepath
Enter choice [1-3]: 1
Enter VM_API_HOST (default: https://api.prod.validmind.ai/api/v1/tracking): 
Enter VM_API_KEY: ***
Enter VM_API_SECRET: ***
Enter VM_API_MODEL: ***

WARNING (/Applications/quarto/share/filters/main.lua:9521) Unable to parse table from raw html block: skipping.

GitHub workflows

These have now been adjusted to accommodate for executing the validation notebooks alongside the development ones:

How it works

Code samples from the PR preview/validate workflow, the functionality has been replicated across all workflows/actions:

1. I added an additional GitHub secret called PLATFORM_VALID_MODEL for the ValidMind Academy validation model.
2. Now the workflows create two temporary .env files: dev.env. & valid.env from the PLATFORM_DEV_MODEL & PLATFORM_VALID_MODEL respectively:

    # If yes then create the dev.env file for use in execution step
    - name: Create dev.env file
      if: steps.filter.outputs.notebooks == 'true'
      id: create_dev_env
      run: |
        touch dev.env
        echo VM_API_HOST=${{ secrets.PLATFORM_API_HOST }} >> dev.env
        echo VM_API_KEY=${{ secrets.PLATFORM_API_KEY }} >> dev.env
        echo VM_API_SECRET=${{ secrets.PLATFORM_API_SECRET }} >> dev.env
        echo VM_API_MODEL=${{ secrets.PLATFORM_DEV_MODEL }} >> dev.env
        cat dev.env

    # If yes then create the valid.env file for use in execution step
    - name: Create valid.env file
      if: steps.filter.outputs.notebooks == 'true'
      id: create_valid_env
      run: |
        touch valid.env
        echo VM_API_HOST=${{ secrets.PLATFORM_API_HOST }} >> valid.env
        echo VM_API_KEY=${{ secrets.PLATFORM_API_KEY }} >> valid.env
        echo VM_API_SECRET=${{ secrets.PLATFORM_API_SECRET }} >> valid.env
        echo VM_API_MODEL=${{ secrets.PLATFORM_VALID_MODEL }} >> valid.env
        cat valid.env

3. The two .envs are passed in as inputs into the composite action for executing the notebook:

    # Only execute the demo notebooks for training if .env files are created
    - name: Execute demo ValidMind for model development and validation series
      if: ${{ vars.ENABLE_DEMO_NOTEBOOK == 'true' && steps.create_dev_env.outcome == 'success' && steps.create_valid_env.outcome == 'success' }}
      uses: ./.github/actions/demo-notebook
      id: execute-demo-notebook
      with:
        dev_env: dev.env
        valid_env: valid.env

4. The composite action checks for the required .env inputs:

inputs:
  dev_env:
    description: "Load the created dev.env file"
    required: true
  valid_env:
    description: "Load the created valid.env file"
    required: true

5. Then executes the notebooks with each respective .env file, temporarily copying it into an .env (without a prefix/filename format) so that the notebooks don't throw an .env not found error:

    - name: Ensure dev.env file is available
      shell: bash
      id: find_dev_env
      run: |
        if [ ! -f "${{ inputs.dev_env }}" ]; then
          echo "Error: dev.env file not found at ${{ inputs.dev_env }}"
          exit 1
        fi

    - name: Execute ONLY the ValidMind for model development series with heap development
      shell: bash
      if: ${{ steps.find_dev_env.outcome == 'success' }}
      run: |
          cd site
          cp ../${{ inputs.dev_env }} ../.env
          source ../.env
          quarto render --profile exe-demo notebooks/EXECUTED/model_development &> render_errors.log || {
            echo "Execute for ValidMind for model development series failed";
            cat render_errors.log;
            exit 1;
          }

    - name: Ensure valid.env file is available
      shell: bash
      id: find_valid_env
      run: |
        if [ ! -f "${{ inputs.valid_env }}" ]; then
          echo "Error: valid.env file not found at ${{ inputs.valid_env }}"
          exit 1
        fi

    - name: Execute ONLY the ValidMind for model validation series with heap development
      shell: bash
      if: ${{ steps.find_valid_env.outcome == 'success' }}
      run: |
          cd site
          cp ../${{ inputs.valid_env }} ../.env
          source ../.env
          quarto render --profile exe-demo notebooks/EXECUTED/model_validation &> render_errors.log || {
            echo "Execute for ValidMind for model validation series failed";
            cat render_errors.log;
            exit 1;
          }

Developer Fundamentals

REGISTRATION PAGE PREVIEW — 🚨 SEE COURSE MODULE PREVIEW DISCLAIMER ABOVE!!!! 🚨

I gave these another proofread/quality-check and edited some minor errors or screen real estate.

External Release Notes

ValidMind Academy

Validator Fundamentals

With the goal of delivering more approachable and comprehensive training, we've overhauled our Validator Fundamentals course into four modules with clear learning outcomes:

1. Module 1 — Using ValidMind for Model Validation
2. Module 2 — Running Data Quality Tests
3. Module 3 — Developing Challenger Models
4. Module 4 — Finalizing Validation Reports

This training series walks you through the basics of using ValidMind to validate datasets and models, from running and logging tests with the ValidMind Library to finalizing your model's validation report in the ValidMind Platform. You'll learn how to use ValidMind to automate testing, populate evidence in your validation report, track finding resolution, and more — register to take the course now!